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Aim and justification for the cocreated program logic 


In order to design, implement, and evaluate organizational interventions, theories 
of change are needed — that is, theories that outline why a certain intervention 
activity would be expected to have an effect on a specific distal outcome (Blamey 
& Mackenzie, 2007). Yet, overall, these types of theories are seldom used in 
organizational interventions (Biron & Karanika-Murray, 2013; Nielsen, 2013). 
This stands in contrast to the abundance of theories like the job demand resources 
(JDR) model and the effort reward imbalance model (e.g., Demerouti et al., 
2001; Siegrist, 1996), which link exposure to factors in the work environment to 
employee health and well-being outcomes (Kristensen, 2005) and therefore may 
guide the content of an intervention. The idea of occupational health interventions 
is generally to improve employee health and well-being through an intervention 
that decreases exposure to demands and/or increases employees’ resources. Thus, 
theories such as the JOR model are helpful in establishing the connection between 
change in job demands and resources and employee well-being. Yet, they are not 
helpful for linking the intervention to the change in job demands and resources. 
This is what theories of change are for. 

Whereas efforts to develop and apply social, organizational, and psychologi- 
cal theories to illuminate how organizational intervention is brought about (e.g., 
Nielsen et al., 2014) are ongoing, the field of evaluation has a long tradition of 
using program logic for the same objective. Program logic, also known as program 
theory or logic models, outlines how an intervention, through its specific interven- 
tion activities, is related to a chain of outcomes, from the most proximal ones to 
more distal ones. This links the design and the implementation of an intervention 
to its evaluation, and thereby, makes the theory of change more explicit (Olsen 
et al., 2012; Rogers, 2008). 
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Program logic has often been constructed by interventionist (i.e., interven- 
tion developers such as consultants or researchers) (Saunders et al., 2005). Here, 
we propose a cocreation process involving multiple organizational stakeholders 
in addition to researchers and consultants. The program logic is then used as a 
guiding framework that runs through the creation of the intervention, its imple- 
mentation, and evaluation. In the business field, the concept of cocreation was 
presented by Prahalad and Ramaswamy (2000), and has spread since. Cocreation 
is one of several terms (e.g., codesign, coproduction, cocare, etc.) stressing that 
design, implementation, and evaluation needs to be a joint venture of the research- 
ers and the organization. Formally, cocreation is an interconnected, recursive set 
of interactions between stakeholders (e.g., managers, employees, researchers, and 
consultants; Payne et al., 2008; Prahalad & Ramaswamy, 2000). Instead of organi- 
zations being passive recipients of services and products (“value”), they are engaged 
in cocreating value, building on their unique perspective and knowledge (Payne 
et al., 2008). Thus, the cocreation process is a way to ensure that the intervention 
process is truly participatory. 

A participatory approach is far from new in the context of organizational inter- 
ventions; on the contrary, it is the recommended approach (Lamontagne et al.; 
Nielsen, 2013; Nielsen & Randall, 2012). It is well known that managers and 
employees are not passive recipients of an intervention, rather they are and should be 
actively engaged in shaping the intervention (Nielsen, 2013). Thus, an organizational 
intervention is not something researchers or consultants can design and implement, 
but something the organization and its members are, to varying degrees, active in 
designing and carrying out (McVicar et al., 2013; Nielsen & Miraglia, 2017). 

In a participatory approach, actors with different kinds of knowledge, skills, and 
perspectives are welcomed to the table. In essence, this process ensures that the 
need to integrate theoretical and practical knowledge is met. The challenge, then, 
is to make sure the differences converge and that all stakeholders contribute to 
creating an intervention that will lead to the target outcomes and that provides the 
best possible match between the different knowledge sources. This includes what 
is known from research, as well as knowledge about the specific organization, from 
different sources within the organization. But how do you do this? 

This chapter outlines a structured process — the cocreated program logic (COP) 
process — for how organizational stakeholders can be engaged in defining inter- 
vention goals and activities and thus forming the program logic together with 
interventionists (researchers or consultants). The program logic can then be used 
to guide the evaluation of the organizational intervention. In this chapter, will also 
present how COP can be used in two different ways: (1) to inform the evalua- 
tion of an intervention where the intervention activities are preset, and (2) to, in 
addition to informing the evaluation of the intervention, also design intervention 
activities. Three objectives form the background for this approach: the necessity 
of a cocreation process in participatory approaches, the need for program logic to 
guide the design, implementation, and evaluation of organizational interventions, 
and the need to link the two together in a structured way. 
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Cocreation to establish ownership and utilization of 
best available knowledge 


As interventions have moved from focusing on changing individual health 
behaviors to target organizational and multilevel structures, the importance of 
engaging stakeholders across the organization has become evident. This engage- 
ment includes having employees and managers across the organization participate 
in change efforts to create a sense of ownership in the organization for the change 
process. In this sense, organizational interventions are less of a time-limited, exter- 
nally induced project and more of an ongoing, continual improvement effort that 
is more closely linked to daily operations. As such, any organizational intervention 
needs to be aligned with organizational visions, goals, and objectives (i.e., verti- 
cal alignment) as well as fitting with daily operations (i.e., horizontal alignment; 
von Thiele Schwarz & Hasson, 2013). Thus, the intervention needs to provide a 
so-called philosophical fit (with the organization’s vision and goals) and a practical 
fit (that is, be possible to do in consideration of possibilities and constraints in the 
organization; Moore et al., 2013). 

The need for alignment and fit means that many stakeholders need to be 
involved in the design, implementation, and evaluation of organizational inter- 
ventions. Stakeholders who bring unique perspectives and sources of knowledge 
about the organization include employees, line and senior managers, researchers, 
occupational health specialists, human resources specialists, change agents (e.g., 
employees with certain areas of responsibility such as safety champions), and/or 
consultants. Employees need to be involved since they are instrumental in bringing 
about change. Line managers are known to be able to make or break an interven- 
tion, but given their role in a hierarchical organization, they are in turn dependent 
on senior managers, who provide (or do not provide) recourses for line managers 
and employees (Hasson et al., 2014). They also bring a wider perspective on how 
an intervention relates to the overall strategies and objectives of the organization. 
Then there are specialists on the relationship between work factors and employee 
health outcomes, as well as specialists on change and evaluation. This can include 
people such as a human resources specialists, consultants, and researchers. Combined, 
these stakeholders bring theoretical and practical knowledge that is essential for 
making sure that each organizational interventions builds on the best available 
evidence from a wide range of sources (that is, not only research evidence). To 
achieve a unified and coordinated change effort, all these knowledge sources and 
perspectives need to be brought together in planning and designing interventions. 
This includes agreeing on the objectives for the intervention, as well as a joint 
understanding of which activities will most likely bring about the desired change. 

Yet, few concrete suggestions and tools exist for how these kinds of processes 
can be realized in practice. For example, the interactions between researchers 
and the organization and its employees have not been particularly well defined 
in previous research — it may be everything from the organization or the employ- 
ees simply accepting the intervention, to the organization and/or the employees 
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having complete ownership of the change process (Kristensen, 2005). Similarly, 
participation may be direct or through representatives. The optimal level of par- 
ticipation is likely to differ between different organizations — one solution that 
fits to all organizations and interventions does not exist. For example, solutions 
may differ between interventions that the organization voluntarily commits to and 
interventions that are launched in response to external demands from changes in 
legislation or national guidelines, or between initiatives that comes from senior 
management and employee-driven changes. Thus, a tool is needed that is flexible 
enough to encompass different types of change processes, yet structured enough 
to provide a clear step-by-step guide for how participation throughout the design, 
implementation, and evaluation of organizational interventions can be achieved. 
Following this, the tool that we present is a suggestion on how a cocreation process 
can be set up and how it can look in practice. 


Program logic: Outlining intervention components 
and target outcomes 


The outcomes resulting from an organizational intervention generally develop as 
a chain of effects (Nielsen & Simonsen Abildgaard, 2013; von Thiele Schwarz 
et al., 2016) linking the components of the intervention to the outcome. For 
example, taking active part in an intervention (exposure and other implementa- 
tion outcomes) aimed at redesigning how work is done (expressed in behavioral 
outcome) may lead to increased job autonomy and improved job clarity (interme- 
diate outcome such as improved psychosocial work environment), which in turn 
increases job satisfaction and work engagement (distal outcomes such as employee 
well-being). That, in turn, may for example decrease turnover (end outcome such 
as organizational outcome). That is, each different outcome logically follows the 
previous one. This chain represents the program logic. It outlines the logic series 
of steps that are necessary for a chain of outcomes to be achieved. 

The program logic, thus, outlines how the intervention is linked to the out- 
comes. This involves clarifying the core components of the intervention and the 
expected consequences of those components. Core components are the activities 
that are essential for the intervention to achieve its outcomes in that without them, 
the intervention will be less effective (or ineffective) (Fixsen et al., 2005). One can 
use multiple ways of outlining the core components. Outlines can be based on 
experience, previous empirical research of effective interventions, or theory. As 
described above, occupational health theories are helpful for guiding which and 
how outcomes (such as psychosocial work environment and employee health and 
well-being outcomes) are related; that is, the later stages of the program logic. For 
the earlier stages, other theories can be useful, including theories explaining behav- 
ioral change; for example, social learning theory and theory of planned behavior 
(e.g., Ajzen, 1991; Thomas et al., 2014) and theories explaining learning; that is, 
pedagogical theories such as constructive alignment (Biggs, 1996) and Vygotsky’s 
zone of proximal development (Chaiklin, 2003). 
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By outlining how outcomes are expected to unfold, program logic also provides 
a framework for evaluation. As different effects follow on previous ones, this sug- 
gests when assessment of the different outcomes (i.e., what to assess) should be done 
in order to capture the effects. More so, the program logic can be viewed as an 
outline of the hypotheses for how the intervention will have its effect on outcomes. 
Having an a priori-specified model for the intervention is particularly important 
for organizational interventions that are not easily evaluated with randomized and 
controlled designs, and that aim to improve distal, multifactorial outcomes such as 
improvements in health (Kristensen, 2005). When outcomes are distal and multi- 
factorial, it is difficult to link the changes or the lack of changes to the intervention. 
Thus, outlining and assessing more proximal, intermediate outcomes may allow 
the footprints of the intervention to be captured. In these circumstances, which are 
common for organizational interventions, the program logic provides an explicit 
and prospective way of studying the relationships between variables. 


Getting everyone on the same page: Backward-moving 
program logic 


Program logic can be developed in many different ways. Traditionally, the process 
involves starting with a predefined intervention where the core components are 
more or less known (e.g., stress management programs, mindfulness trainings, and 
leadership trainings). The intermediate and increasingly distal outcomes are then 
outlined in sequential order (Saunders et al., 2005). In contrast, the starting point in 
COP is at the outcomes: what goals are to be achieved? The logic is that in organi- 
zational interventions, achieving the outcome is more central than implementing a 
specific intervention. This approach is similar to that used in quality improvement 
research (Reed et al., 2014). 

The backward-moving program logic is in line with the dynamic integrated 
evaluation model (DIEM; von Thiele Schwarz et al., 2016). This is an evaluation 
model for interventions that are dynamic (i.e., changing over time) and integrated 
(i.e., piggybacks on existing processes and structures in the organization) (von Thiele 
Schwarz et al., 2016). DIEM covers the intervention design, its implementation, fur- 
ther improvements of the intervention, and evaluation. The first four steps in DIEM 
cover the design phase, including decisions on objectives and target outcomes. COP 
can be used as a practical tool in these steps to define the intervention goals and 
intervention activities. After arriving at an intervention prototype (i.e., what activi- 
ties we think are suitable), that is, the best current idea about how the intervention 
will look in the current context, the following DIEM-steps (five to eight) cover 
the implementation of the prototype and the potential revisions to it. These steps 
involve the continuous evaluation of how the intervention works in practice using 
data as a basis for potential revisions. Furthermore, evaluation involves the measure- 
ment of the intermediate and distal outcomes. COP is also a tool to define what 
type of outcomes should be measured and what are the optimal time points for these 
measurements. Thus, COP is central to several phases of an intervention. 
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Potential advantages of COP 


Although it is possible for an interventionist or researcher to develop a program 
logic without involving other stakeholders, one of COP’s defining features is that 
the program logic is cocreated amongst the group of stakeholders. This has a num- 
ber of advantages. 

First, cocreating the program logic may help get everyone on the same page, 
that is, to form a common understanding (a shared mental model) of the inter- 
vention (Nielsen & Randall, 2013). This decreases the risk of friction once the 
intervention is implemented. It may also decrease the risk of perceptual distance 
between stakeholders, that is, that different actors have different ideas and expec- 
tations about the intervention, its aim, goals, and needs (Hasson et al., 2016). As 
perceptual distance has been suggested to have a negative impact on the implemen- 
tation of interventions as well as their outcome, decreasing perceptual distances 
concerning the intervention upfront may promote the successful implementation 
of the intervention (Hasson et al., 2016). 

Second, cocreating the program logic may help build commitment and engage- 
ment. The necessity of having people across the organization on board is well 
known. By inviting those who will have opinions about the intervention, and 
allowing them to be able to influence whether it is implemented or not, a cocreated 
program logic means that those who can make or break the intervention will have 
invested time and intellectual capital in the development of it. This may increase 
the likelihood that they will assume ownership over the intervention as it unfolds. 

Third, the program logic is likely to be more accurate if people with different 
knowledge sources and viewpoints have been involved in cocreating it. Having 
access to different knowledge sources (e.g., both theoretical knowledge about the 
intervention and practical knowledge about how things works in the organization) 
contributes to this. 

Fourth, not only will the program logic be more feasible, the fit between the 
intervention and the setting where it is implemented is likely to be improved. 
The concept of intervention fit includes two interdependent dimensions: envi- 
ronment-intervention fit and person-intervention fit (Randall & Nielsen, 2012). 
By cocreating a program logic, constraints and opportunities in the organizational 
environment that may affect the intervention can be made explicit and managed 
by matching the intervention components to the needs in the organization, and 
(if needed) intervention components can be added aiming at managing obstacles 
for change. The cocreated program logic along with the adapted intervention can 
improve the perceived appropriateness of the intervention in the current environ- 
ment (von Thiele Schwarz et al., 2016). Similarly, for those individuals involved 
and for those groups of employees they represent, person-intervention fit may be 
improved since the participants are likely to ensure that the intervention benefits 
them and those whom they represent. With a cocreation process with multiple 
stakeholders involved, this increases the chance that multiple viewpoints and needs 
will be addressed. 
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Lastly, by cocreating program logic the groundwork for the evaluability of the 
intervention is laid. The program logic forms a map that outlines what and when 
to evaluate and what data to collect. Since cocreating program logic helps establish 
a shared perception of the intervention activities and target outcomes as well as the 
mechanisms between them, those involved will understand the conclusions drawn 
from an evaluation, and thus, be more likely to embrace the findings (Blamey & 
Mackenzie, 2007; Leviton et al., 2010). 


The COP process 


The COP process builds on a structured methodology developed in higher edu- 
cation called adaptive reflection (Savage, 2011). Adaptive reflection combines 
the pedagogical theories of constructive alignment (Biggs, 1996), Bloom’s tax- 
onomy (Bloom et al., 1956) and Kolb’s experiential learning process (Kolb, 
1984). In constructive alignment applied in higher education, the outcomes 
that students are intended to learn are the starting point, and learning activities 
are aligned with these outcomes. It is important to note that the learning out- 
comes are expressed in active terms (e.g., describe, reflect, use, exemplify, etc.) 
that describe what performance is needed to achieve the outcomes. The active 
verb also indicates what kinds of learning activities are most suitable. For exam- 
ple, if the learning outcome is to apply something (e.g., give feedback), then the 
learning activity should provide opportunities to practice giving feedback. Bloom’s 
taxonomy is a model covering a list of learning activities presented hierarchi- 
cally, going from surface representations of learning (e.g., recognizing) through 
learning that reflects an increasingly greater ability to elaborate and use this 
knowledge (e.g., through describing, applying, analyzing, evaluating, and creating). 

In the COP process, similar to adaptive reflection, the process of creating a con- 
structive alignment between activities and outcomes is done with the stakeholders. 
They are led through a step-by-step process, outlined in Figure 2.1 (left-hand 
side). Figure 2.1 includes a description of the general steps as well as the ways the 
COP process was used in the two cases presented in this chapter, illustrating the 
flexibility of the tool. 

The first step contains individual reflections over what it would look like if the 
outcomes of the intervention were achieved (see case descriptions for example). By 
reflecting individually at first, the benefits of having various perspectives is repre- 
sented without risking anchoring effects and conformities, which regularly happen 
if one opinion is voiced before everyone has been able to contribute (Kahneman, 
2011). Also, by asking the participants to build on their own experiences and then 
reflect and elaborate on them, the idea is to trigger a process of experiential learn- 
ing (Kolb, 1984). The thoughts are documented on Post-it notes. In the second 
step, the Post-it notes are compiled and the participants asked to sort them into 
meaningful categories — initially under silence. This is, again, to allow individual 
interpretation, to avoid premature compromises, and to avoid the influence of 
power structures that may cap the knowledge becoming available in the group 
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Case 2- Designing intervention | 


Steps | General COP Activity ) | Case 1- Guiding evaluation | i and guiding evaluation 


What behaviours, skills, and attitudes do [the 
Step 1 target individual] in [our organization] need to 
achieve [overall intended outcome ]? | Step 1 Step1 Step 1 | | Step 1 | Step 1 | Step 1 | 

Brainstorm, document on post-it notes 
(individually) 


| D | | $ 4 D 


What clusters of outcomes to we intended this 


Step 2 intervention to lead to? Sort post-its into | Step 2 | | Step 2 | | Step 2 | Step 2 | Step 2 | [Step 2) 
| meaningful categories (groups, in silence) 
| | | | l l | 
How can the target outcomes be described? 
Step 3 Name each category, add the appropriate active | Step d Step 3 Pa Step 3 | Step 3 | Step 3 | | Step 3 | 
Kee verb so that a full sentence is formed (group) 
Which activities are best suited for achieving 
REES the intended outcome? List possible 
Step 4 intervention activities and document them on 
— post-its (individually). Compile them and Step 4 


match the activities to the intended outcomes, 
minding the active verb (group) 


What contextual factors will make or break 4 d | 


this intervention? Brainstorm opportunities 

Step 5 and challenges that will make or break the Step 5 Step 5 | 

— intervention in this specific context, document 
on post-its (individually) 


FIGURE 2.1 The COP process 


(Savage, 2011). In the third step, each category is named and a suitable active verb 
is identified, forming a full sentence describing the target outcome in active terms. 
To help with this process, a list of active verbs can be presented (e.g., compare, 
analyze, inform, or reflect). In the fourth step, intervention activities are listed 
through individual brainstorming and then matched to the target outcomes. The 
final step in COP is to brainstorm factors that could make or break the interven- 
tion in this specific setting. This is also documented on Post-it notes individually. 


Two cases of COP in action 


In the following section, two cases illustrating how COP can be used in practice 
are described and discussed. First, a case where COP was used to create a common 
understanding of the objectives and to guide evaluation for a network-based learn- 
ing model aiming to improve eHealth utility in a large health care organization is 
presented. The second case outlines how COP was used in a multilevel interven- 
tion including a first-line manager training intervention and a supporting senior 
management intervention (see also Chapter 9 for details on the senior manage- 
ment intervention). These cases are chosen to (a) outline how COP can be used 
to guide evaluation when the intervention is predefined (case 1) and (b) to inform 
the design of an intervention as well as guide its evaluation (case 2; see Figure 2.1, 
right-hand side). For each case, the contextual setting for the intervention and the 
background for the intervention are presented before explaining how COP was 
used. Then the results of the COP process are presented. We end each case by 
presenting some of the possibilities and limitations of using COP in the specific 
context of the case. 
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Case 1: Defining objectives to guide evaluation of an eHealth 
skills development intervention 


Setting and background 


This case focuses on a participatory employee skills development program. It was 
conducted in a regional health care organization in Stockholm County, Sweden, 
that included primary care, psychiatric care, habilitation, and rehabilitation divi- 
sions, as well as the five largest hospitals. The head of the primary care division, 
also responsible for the entire development program, contracted us in the autumn 
of 2015 to evaluate the intervention. The authors of this chapter had previously 
evaluated a similar intervention (Augustsson et al., 2017). 


Intervention participants 


All employees (i.e., 44,000 staff members) are expected to participate in the 
skills development program during 2017—2019. The intervention is a partici- 
patory skills development program consisting of a series of cross-professional 
workshops aimed to improve employees’ skills in and use of information and 
communication technologies (ICT), clarify roles regarding ICT, decrease 
demands in general, increase work satisfaction, and also in a longer run strengthen 
employees’ employability. 

The intervention has a network design and is led by an internal project manage- 
ment team supported by consultants specialized in process evaluation. They work 
together with a group of process instructors to design the themes and materials for 
the workshops that form the content of the intervention. The process instructors 
are health care staff working in the organization and thus they are familiar with 
the organizational context. The process instructors’ task is also to coach development 
leaders. These are employees who have volunteered to lead the workshops that are 
conducted at each unit. They are also expected to act as embedded change agents. 
Thus, the workshops are led by different development leaders across the different 
units, but they use the same structure, themes, and materials. 

The participatory design of the workshops entails active participation of all 
employees in discussions, reflections, and practical exercises rather than didactic 
teaching. This also means that the questions discussed and the amount of time 
dedicated to different parts of a theme are allowed to differ somewhat between 
different sessions. Approximately 10 employees participate at each workshop ses- 
sion and every workshop theme is repeated until all employees in a unit have had 
the opportunity to participate. Overall the intervention includes three workshop 
themes. Each workshop lasts between 2.5 and 3 hours. 


COP: Cocreated program logic to guide evaluation 


In this case, COP was used to guide the intervention evaluation (agree on out- 
comes and the logic relationships between outcomes). This was done during one 
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workshop in September 2016. Participants were process instructors (e.g., internal 
change agents, n = 9) and the project management team (n = 3). The focus was on 
explicating the outcomes of the development program for the three main groups 
in the network model: the employees, the development leaders, and the process 
instructors. The goal with the COP process was to facilitate the development of 
a shared understanding of the goals of the intervention. In particular, because the 
intervention was set up as a network model, the aim was to make clear the con- 
nections between actions and outcomes across the network. The defined outcomes 
then guided the intervention evaluation. 

Rather than running the COP steps separately for each group (i.e., employees, 
developmental leaders, and the process instructors), we ran them in parallel during 
the same workshop as outlined below (see also Figure 2.1). 

Step 1. Following the general COP process outlined above, the first step 
involved individual work. We tailored the process slightly by repeating the first 
step so that outcomes for each group (employees, development leaders, and pro- 
cess instructors) in the network were covered. First, all participants were asked 
to consider the outcomes for employees. The original COP question was “What 
behaviors, skills, and attitudes do employees need to have after the intervention 
so that the project goals are met?” This was tailored based on input from the 
participants so that it would reflect the participatory process of the intervention. 
Participants were therefore asked to think individually about the question “What 
behaviors do employees need to engage in when participating in the intervention 
so that the workshop goals and the overall project goals are met?” The participants 
wrote their thoughts on Post-it notes (one thought per note). The Post-it notes 
were compiled and put aside for the moment. 

Next, step 1 was repeated focusing on development leaders, asking “What 
behaviors, skills, and attitudes do development leaders need to have for the project 
goals to be met?” Again, thoughts were written down on Post-it notes, which 
were compiled and separated from the employee notes. After this, the same pro- 
cedure was repeated with focus on the outcome for the process instructors. At 
this stage, the question was “What additional behaviors, skills, and attitudes (than 
those already mentioned for development leaders) do the process instructors need 
to show for the project goals to be met?” The participants wrote their thoughts on 
Post-it notes, which were gathered and compiled separately from the two other 
levels’ outcomes. 

Step 2. Following the COP process, the Post-it notes were clustered within 
each group according to common themes. Now, the process for each of the actors 
in the network model (employees, developmental leaders, and process instructors) 
was run in parallel. The participants were divided into three groups. The partici- 
pants worked in silence to sort the Post-it notes into themes. 

Step 3. Once the groups were pleased with their themes, they were asked to 
discuss and to create headlines for the clusters. The clusters of Post-it notes for the 
developmental leaders dealt with the themes of being able to collaborate and com- 
municate with all relevant stakeholders; having thorough understanding of people, 
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the work conducted at the unit, ICT, and the organization; showing good seminar 
leadership skills; being able to develop in the role of seminar leaders; being able to 
learn from others; being able to engage all employees in the discussion (being able 
to listen, encourage those being more quite, being able to deal with those with 
strong emotions); and being confident in the participatory process, benefits of the 
training, and the organization being able to support them. 

In an extension of step 3 of the original COP process, each group presented 
the headings to the rest of the participants. A discussion about the outcomes was 
facilitated by the researchers in order to create a common understanding of the 
headings and to get a deeper understanding of the themes. 

The participants particularly highlighted that it became clear to them how the 
outcomes of the organization (on the employee level) were determined by how 
the developmental leaders were functioning, and how this, in turn, was dependent 
on how the process leaders acted. The groups also noted that the main themes were 
related to leadership and facilitation rather than content knowledge about eHealth. 
This was taken as an indication of the need to make sure this was reflected in 
the intervention activities for developmental leaders, which were initially focused 
more on eHealth content than change leadership. Due to the limited time that 
could be set aside for the workshop, this step did not involve finding active words 
as proposed in the original COP process. 

Step 4. As this was an intervention where the intervention activities were already 
designed, the fourth step of listing intervention activities and matching them to the 
target outcomes was not applicable. 

Step 5. The last step was an individual brainstorming activity to identify the 
organizational context that may influence the possibility of achieving the defined 
outcomes. The question the participants reflected upon was “What organizational 
context do development leaders need in order to succeed as seminar leaders?” 
Participants were instructed to think particularly about things that were feasible 
in their context, rather than visualizing the ideal organization where time and 
resources would be infinite. Again, thoughts were documented on Post-it notes. 
Aspects that were named was having mandate and support from their line manager 
and the process instructor were clear descriptions of what is expected of them, a 
general understanding in the organization that this participatory intervention is pri- 
oritized and linked to overall organizational objectives, feedback on performance 
from the project management team and the process instructors, capability to lead 
seminars and training in being able to do so, and practical aspects such as enough 
time allocated to the task as seminar leaders. This information was used to broaden 
the understanding among participants of the interconnectedness of the change ini- 
tiatives within the broader organizational context as well as to guide the evaluation 
(e.g., suggestions for possible mediators and moderators of change). 

Finally, the information from the COP-workshops was used to inform the 
choice of measurement. Items of established scales were identified and mapped on 
the target outcomes to ensure that the evaluation reflected. 
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Lessons learned when COP is used to inform evaluation: A 
self-evaluation 


In this case, COP was used to guide the evaluation of an eHealth development 
program, and we found that the process did so by providing valuable information. 
The clustering of outcomes showed which constructs needed to be included in the 
evaluation. The individual Post-it notes then helped ensure that the items reflected 
the specific meaning that the stakeholders attached to the construct. This can be 
thought of as a simple way of tailoring the measurement to the specific context, 
which has been suggested to be critical when evaluating organizational interven- 
tions (Nielsen et al., 2014). This grounds the evaluation in the context where it is 
conducted, increasing the likelihood that it will be meaningful for the organization 
and that it will be sensitive to the changes it aims to measure. In this case, the eval- 
uation is still ongoing and thus, we do not yet have data on how the evaluation has 
been experienced so far; it has been helpful to be able to fall back on that content 
of the evaluation that was cocreated., helping to pave the way for acceptance of the 
results of evaluation, whatever they may be (e.g., evaluability; Leviton et al., 2010). 

From observations of the workshop and the conversations that took place, it 
was clear that inviting organizational stakeholders engaged in the intervention to a 
joint workshop gave the group an opportunity to discuss the project goals in more 
concrete terms. It seemed to facilitate the development of a common understand- 
ing of the outcomes. This may be particularly important since the group included 
both the project management team and the process instructors, that is, those over- 
all responsible for the intervention and those who had been asked to join so as 
to inform intervention activities and to facilitate the delivery of the intervention. 
Having a common understanding of the goals is crucial because the intervention 
builds on a network model where each instructor and seminar leader is supposed 
to use the same material and themes as a basis and thereafter allow certain varia- 
tions in the actual execution of the participatory workshops. Being on the same 
page is likely to contribute to the different nodes of the network moving in 
the same direction. 

Using outcomes on three levels for employees, development leaders and for 
process instructors, seemed valuable both for us as evaluators and for the organi- 
zational stakeholders. For us, it helped explicate the logic model linking activities 
in different parts of the network to the end (employee) outcome. By working 
through the outcomes for the different actors, the links between them became 
evident also for the participants in the workshop. For example, the participants 
noted that many more leadership-related tasks were involved in their picture of 
a successful developmental leader than, for example, ICT competences. In fact, 
some participants expressed a revelation about the fact that the role was much 
more of being a change agent than they had realized before. In addition, they 
were somewhat surprised to realize their own roles as process managers had an 
important function in forming a context that would give developmental leaders 
opportunities to lead, which in turn would be necessary in order for the outcomes 
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on the employee level to be achieved. Overall, the importance of the process parts 
of the intervention, particularly the participatory approach, became clear through 
the COP process, suggesting that intervention activities targeting these areas may 
be needed. 

Nevertheless, designing the intervention activities was beyond the scope of the 
evaluators’ assignment as the intervention activities, overall, were preset. This is 
clearly a disadvantage, as relevant information became readily available through 
the workshop despite that the step specifically designed for this (step 4, Figure 2.1) 
was skipped. Therefore, even though the COP workshop gave input to the project 
management group, the usefulness of COP would have been even greater if the 
process also included informing the design of the intervention. 

Working in parallel with the three different groups was a time-efficient way of 
covering a lot of material: the workshop only lasted 2 hours. It also saved the par- 
ticipants from the tedious repetition of going through the same process three times. 
Nevertheless, this may introduce a risk of developing different, rather than shared, 
understanding in the three groups. We tried to mitigate this risk by first having 
everyone contribute data for all three actors (step 1) before splitting into groups, 
and then, after the third step, letting the groups present and discuss each group’s 
findings. Nevertheless, we would suggest adding another hour to the process to 
let this discussion take its time. This would also allow time to turn the headings of 
the clusters into meanings with active verbs, which was skipped because of time 
restraints. Adding that would allow more detailed analyses of the skills, knowledge, 
and behaviors than just description of the main themes. 


Case 2: iLead — a multilevel intervention to foster 
implementation leadership 


Setting and background 


The second case focuses on a multilevel intervention that aimed to increase imple- 
mentation leadership among line managers. This intervention was conducted in 
one division of the regional health care organization, with practices spread out 
throughout Stockholm County, Sweden. Senior management contacted us for 
help in the implementation of a working method that aims at facilitating and mak- 
ing care planning more effective. This implementation process had been on-going 
for approximately 18 months with a variety of setbacks. 

Approximately 700 employees work in the division, which is divided into five 
thematic sections. Further, the sections are divided up into units, which are led 
by 33 line managers. The senior management group consists of nine members, 
who decided to invite all line managers to participate in the intervention. During 
the planning process, the senior management group also realized that they, too, 
needed to develop their own knowledge and skills about implementation and how 
to lead implementations, and thus, the line leadership intervention was comple- 
mented by a supporting intervention specifically for the senior management group. 
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Some of the content was similar, but overall, the senior management intervention 
focused on improving their ability to provide line managers with an optimal con- 
text to perform their implementation leadership. For further information about 
the project in general, see Richter et al. (2016), and for the senior management 
intervention in particular, see Chapter 9. 


COP and intervention participants 


Five national experts in implementation and leadership training (consults or 
researchers in the area) participated in the COP process preceding the intervention. 
Thirty-one line managers participated in the COP process and the intervention (one 
was unable due to ongoing organizational restructuring). The majority of participants 
were female, representing the general gender composition in health care. All partici- 
pating managers chose a current implementation that was relevant for them to work 
on during the leadership intervention. In addition, the whole senior management 
group participated in the COP process and in the senior management intervention. 


Using COP to design the intervention and guide evaluation 


To design the intervention (the goals and the activities) and to guide the evaluation 
(agree on outcome and the causal relationships between outcomes), three separate 
COP workshops were conducted from winter 2015 to spring 2016. 

The first COP workshop was conducted with experts. In addition to inform- 
ing the design of the intervention and the evaluation in general, the specific 
goal was to get a coherent expert opinion about appropriate intended outcomes 
of implementation leadership training interventions and to identify intervention 
activities that where constructively aligned with those outcomes. Following the 
COP process outlined above, the followings steps were taken (see Figure 2.1, 
right-hand side). 

Step 1. The experts were instructed to think individually about the question of 
“What behaviors, skills, and attitudes do managers need to lead an implementa- 
tion?” They wrote down all thoughts on Post-it notes (one thought per note; see 
Table 2.1 for examples). 

Step 2. The notes were compiled, and the experts worked together to sort them 
into meaningful categories, initially under silence. 

Step 3. The experts then created headings for each cluster. These headings were 
iteratively revised until they contained an active verb, forming a full sentence that 
described an intended outcome of the intervention. To help with this process, 
the group was presented with a list of verbs. They were also asked to consider the 
level of proficiency that each verb represented in relation to knowledge, learning, 
and skills. In this way, the headings form the target outcomes for the line manager 
intervention (see Figure 2.2). 

Step 4. To arrive at a list of intervention activities that matched the intended 
outcomes, the experts were asked two questions. First, they worked individually 


FIGURE 2.2 The expert group in the process of matching intervention activities to the 
intended outcomes 


FIGURE 2.3 The expert group reflecting on the identified the learning outcomes 
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generating answers to “Which activities have worked well in previous interven- 
tions?” The answers were documented on Post-it notes. These were compiled 
and the group jointly engaged in answering the second question: “Which of these 
learning activities would fit in the current intervention to achieve the target out- 
comes?” They matched the appropriate intervention activities to the intended 
outcomes (Figure 2.3). This included considering the verb (level of proficiency 
of knowledge and skill) and making sure the intervention activities were con- 
structively aligned with the outcome. For example, the intervention outcomes 
“knowledge about relevant theoretical models” and “setting the implementation 
into a larger context” were intended outcomes that were identified by the experts. 
The ambition was also to include a variety of intervention activities such as how 
knowledge about the relevant theoretical models can be brought about (e.g., using 
short inspirational films, paper and pen exercises, etc.). See Table 2.1 for examples 
of identified intervention activities. 

Step 5. Because the experts were not familiar with the specific context where 
this intervention would be implemented, this step was not applicable. 

The second COP workshop targeted the 31 participating line managers. Their 
process involved steps 1-3 and 5. 

Step 1. The first-line managers were asked to respond to the question “What 
behaviors, skills, and attitudes do managers in our organization need to lead an 
implementation?” Thus, they were asked to respond to a similar first question 
as the experts, but specifically considering leading implementation in their own 
organization. Similar to the experts, the line managers were instructed to generate 
as many Post-it notes as possible. See Table 2.1 for examples. 

Step 2. Similar to the general COP process and that used in the expert group, the 
Post-it notes were compiled and then sorted into meaningful clusters, in silence. 
Because this was a larger group, this was done in groups of five to seven persons. 

Step 3. Each smaller group was then asked to find headings for each cluster 
using active verbs. Compared to the expert group the line managers did not relate 
the active verbs to the level of proficiency that each verb represented. Here an 
adaptation was made to fit to time constraints for this workshop. 

Step 5. Going directly to step 5, the managers were asked “What context/sur- 
rounding do line managers in our organization need in order to become good 
implementation leaders?” They were instructed to think particularly about things that 
were feasible in their organizational context, rather than visualizing the ideal organi- 
zation where time and resources are infinite. Examples are presented in Table 2.1. 
For the sake of time, the line managers did not proceed with steps 2 and 3 for this 
question. Also, in contrast to the first question, which aimed at creating a common 
understanding of what implementation leadership is, the second aimed to provide 
input on what supporting activities would be needed, including informing the con- 
tent of the senior management intervention. For example, it became evident that the 
managers needed clarity about the time frame for the implementation and that they 
longed for a more effective dialogue between senior and line managers. 
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Because steps 2—3 were done in smaller group, the line managers spent 5 min- 
utes towards the end of the workshop looking at the headings and Post-it from 
the other groups. This was done to give them a sense of which themes the other 
groups had identified. 

The third COP workshop was conducted with the senior management group. 
It followed the same steps as the process for line managers answering the question 
“What behaviors, skills, and attitudes do first-line managers in our organization 
need to lead an implementation?” They silently wrote down answers on Post-it 
notes, sharing the notes and sorting them in silence, then deciding on the head- 
ing for each cluster. Results of this workshop were similar to the results from the 
workshop with the line managers (see Table 2.1). 

After the workshops, the senior management group and the line managers 
received a transcript of notes from their respective workshops. This was done as a 
memory aid and to give them input for further reflection. The senior management 
group also received transcripts from the first-line manager workshop. 

Once the three workshops had been conducted, the next step involved creating 
a program logic. Here we added another source of information, namely scientific 
literature and theory on leadership and implementation and pedagogical principles. 
A scoping review on these topics was undertaken, particularly looking at the theo- 
retical underpinnings, content, and pedagogical principles of published leadership 
interventions. The results from the COP workshops as well as from the literature 
review were mapped and outlined in a logic model. The researchers conducted 
this in an internal workshop (Figure 2.4). A brief overview of the program logic 
can be found in Table 2.2. Overall, the results from the COP process were well in 
line with the scientific literature and theory. The intervention activities identified 
by the expert group also largely overlapped with pedagogical approaches that have 
previously been used in leadership interventions. 

The program logic was then used to guide the evaluation. This was done in an 
iterative fashion whereby items of established scales measuring constructs relevant 
to the intervention goals were mapped on the intended outcomes to ensure that 
the identified issues were covered in the evaluation. This process highlighted the 
need to develop a scale that specifically captured implementation-specific, full- 
range leadership (Mosson et al., forthcoming). 


Lessons learned when COP is used design the intervention: 
A self-evaluation 


In this second case, COP was used both to inform the design and the evaluation 
of the intervention using a series of COP workshops targeting different stakehold- 
ers (e.g., experts, line managers, and senior management). Overall, this approach 
seemed feasible to elicit the information needed to create a program logic that 
was contextualized to the organization. It also seemed a feasible way to foster a 
shared understanding of the goals of the intervention as well as the logical links 
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FIGURE 2.4 The researchers map the results from the COP process on theoretically 
and empirically derived concepts 


TABLE 2.2 Brief version of the program logic 


Core components 


Immediate impacts 


Short-term impacts 


Distal outcomes 


e Short lectures 
Work with a 
practical case e.g. 
action plan & 
sustainability plan 

e Reflection in 
small groups and 
individually 
Role-play 

e Feedback from 
employees, fellow 
participants 

e Try new leadership 
behaviors — work 
between the 
workshops 

e Booster email 
between the 
workshops 


Increased 
knowledge about 
implementation 
leadership and 
implementation 
models in general 
Increased 
knowledge about 
the specific 
implementation 
process 

Increased 
understanding about 
reactions to change, 
motivation 
Increased self- 
efficacy to lead 
implementation 
Improved ability 
to structure 
implementation 


Improved skills e 
and capacity 

to handle 
resistance, listen 

to employees e 
More frequently 
express trust, 
communicate H 
change in an 
understandable 

and meaningful e 
way 

Setting clear 

goals, monitor 

and give feedback 
more frequently 
Create 

motivation to 
implement 

Provide increased 
direction 


Increased 
implementation 
of the 
guidelines 
Improved 
implementation 
climate 
Improved 
work-related 
wellbeing 
Improved 
productivity 
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between the content of the intervention and the objectives (target outcomes) of 
the intervention. 

One of the advantages of using a series of workshops with the different stake- 
holders was that it allowed the COP process to include only the steps most 
relevant for each specific group. All did steps 1-3 answering the question of “What 
behaviors, skills, and attitudes do managers in our organization need to lead an 
implementation?” But the experts subsequently focused on intervention activities 
and linking them to outcomes, and line managers added information on contextual 
factors that could make or break the intervention. 

The input from the experts (e.g., researchers and consultants) helped to ensure 
that all-important aspects (both practical and theoretical) in the design and evalu- 
ation of the leadership intervention were considered. Thus, this process allowed 
research and practice to be combined, incorporating multiple knowledge sources. 
As the experts had practical experience of working with complex interventions 
and leadership development, they were able to contribute with a practical per- 
spective both in terms of what they felt managers needed to know and do and 
also of “what works” as intervention activities. By inviting experts working in 
different fields and using different learning approaches, a variation of perspectives 
was considered that might not have been included if only relying on literature 
reviews. Nevertheless, the expert group was a convenience sample of experts 
who were well known to the researchers, and the width of experience could 
have been even more diverse, for example, by including experts with other 
disciplinary backgrounds. 

Whereas the experts’ input helped incorporate the theory and practice of 
leadership development, the line managers helped contextualize the interven- 
tion by describing the influence of context. By doing so, they provided valuable 
information about what should be included in a supporting intervention, and it 
was essential for creating the content for the senior management intervention 
(Chapter 9; von Thiele Schwarz et al., 2016). The Post-it notes on the contextual 
challenges were categorized by the interventionists and a summary of the relevant 
categories was sent out before each senior management intervention workshop 
to increase the senior group’s readiness for the workshop and purvey the sense of 
urgency of the topic. 

Inviting line managers to the COP process had additional advantages. First, 
it was a way to get to know the participants and for them to get to know the 
researchers. Therefore, the workshop, which was the first contact with the line 
managers, was important to building a trusting relationship. Second, it was a way 
to build a common understanding amongst the line managers concerning what it 
meant to lead an implementation. It provided line managers with an opportunity 
to reflect on their work and role together with colleagues. They also received tran- 
scripts of the notes and headings from the smaller groups. The fact that the different 
groups had generated very similar topics further conveyed that the perceptions to 
a large degree were shared. Third, the workshop gave the managers the possibility 
to reflect upon their role as implementation leaders and mentally preparing them 
for the role they would be asked to take during the intervention. Although we 
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lack data to support this claim at this point, we speculate that this may be a way to 
increase readiness for change. 

This case also involved a separate COP process for senior management. For 
the researchers, this workshop was an important source of information, as it made 
explicit the expectations that senior management had on what the line managers 
needed to improve, which in turn made the expectations on the intervention and 
the interventionists explicit. Based on observations and comments from the senior 
managers during the workshop, the COP process also seemed to facilitate a better 
understanding of the complexity of implementation leadership. This provided a 
starting point to build on for the senior management intervention, as it clarified the 
demands the line managers were facing and how they can be assisted in their role 
as implementation leaders. Lastly, the COP process also meant that senior manage- 
ment got a sense of involvement and investment in the intervention. The COP 
process may be one way to increase the buy-in that is so important for the success 
of any organizational intervention. 

The workshop with the senior management took 2 hours, but could very 
well have been expanded to also let senior management reflect on their own 
role in the change process. That would have been another source to shape the 
content of the supporting senior management intervention and should definitely 
be introduced if the senior management intervention is the primary rather than 
supporting intervention. 

In this case, the different sources of information from the three stakeholder 
groups converged. This helped convey a sense of shared purpose that was par- 
ticularly important given that the organization had experienced difficulties 
related to the implementation during the years preceding this intervention. It 
also made it seamless for the researchers to put the program logic together. Yet, 
there may not always be convergence between stakeholder groups. In these 
cases, the COP process will help illuminate any perceptual distance that may 
exist between different stakeholders. Such discrepancies will have to be man- 
aged, and to do so, it may be worthwhile to amend the COP process to also 
include more shared sense-making, similar to how it was done in the first case 
in this chapter. 

The results from the COP process also largely converged with previous 
research and leadership theory. This could be interpreted as the process being 
superfluous but on the other hand, we believe this demonstrates the validity of 
the method (as well as the validity of the theories). From a research perspec- 
tive, this means that we still could build the intervention on theory. Yet, we 
still received all the benefit of the cocreation process. Participants themselves 
had generated the content, likely increasing the sense of fit and relevance as 
well as ownership compared to a scenario in which the research team had pre- 
sented a predefined solution to them. We believe that this can increase the 
person-intervention fit as well as the organization-intervention fit, which pre- 
viously has been identified as a crucial factor to succeed with an intervention 
(Randall & Nielsen, 2012). 
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Conclusions 


In this chapter, we present a structured process whereby organizational stakeholders 
and researchers are engaged in designing intervention activities and/or to inform 
the evaluation by outlining the objectives and outcomes, thereby cocreating a pro- 
gram logic. We illustrate it using two cases. The first shows how the COP process 
can be used to guide evaluation and support the development of a shared under- 
standing among stakeholders for a predefined intervention. The second shows how 
COP can inform the design of the intervention, in addition to guiding evaluation. 
In both cases, we perceived the process to be immensely helpful to ensure a thor- 
ough work-through of the program logic as well as strengthen the collaborative 
relationships with the organization where the interventions were set. 
The advantages of the current approach were as follows: 


e The process provided a structured approach to integrating theory and prac- 
tice. On the one hand, the process validated the relevance of theories linking 
participatory approaches and leadership to outcomes. On the other hand, it 
contextualized the theories, tailoring them to the needs of the organizations 
and describing them in the words of the participants. 

e The process was flexible enough to allow changes to be made in response to 
needs expressed by participants (case 1). It also provided a sufficient balance 
between structure and flexibility to allow different stakeholder groups to focus 
on the steps most relevant for them (case 2). 

e The process was active and engaging. The participants expressed that they 
enjoyed the workshops—they were perceived as engaging, fun, and thought 
provoking. They appreciated the practical approach rather than merely dis- 
cussing issues. Thus, as a bonus, the positive experience helped increase 
positive expectations for the coming intervention. 

e The COP process worked equally well with the different stakeholder groups— 
managers and change agents in multiple levels of the organizations. It also 
worked both for a predefined intervention and an intervention where the only 
the main form (a leadership training intervention) was predefined. 

e Through the COP process, the stakeholders help delineate how the objectives 
of the organization could be achieved and what the target outcomes might 
be. In the discussions around the headings, the participants also touched upon 
the prioritization of outcomes and activities, which informed the research- 
ers about the activities that were believed to have the greatest impact, to be 
most changeable, and to have the greatest possibility for positive spillover, 
issues that have been described as essential for matching an intervention to an 
organization (Michie et al., 2015). 


Yet, based on our learning from the two cases, we propose two revisions to 
the COP process. These are summarized in Figure 2.5. The first is an addition of 
a sixth step. Similar to earlier in the COP process, this step involves clustering the 


GA von Thiele Schwarz, Richter and Hasson 


Steps General COP Activity ) 
ra 


What behaviours, skills, and attitudes do [the target individual] in 
Step 1 [our organization] need to achieve [overall intended outcome]? 
Brainstorm, document on Post-it notes (individually) 


18 


What clusters of outcomes to we intended this intervention 
to lead to? Sort Post-its into meaningful categories (groups, 
in silence) 


Step 2 


How can the target outcomes be described? Name each 
Step 3 category, add the appropriate active verb so that a full 


= od L sentence is formed (group) 
Which activities are best suited for achieving the intended outcome? 
Sten A List possible intervention activities and document them on post-its 
P (individually). Compile them and match the activities to 
the intended outcomes, minding the active verb (group) 
What contextual factors will make or break this intervention? 
Step 5 Brainstorm opportunities and challenges that will make or break the 
G intervention in this specific context, document on post-its (individually) 
What clusters of factors will make or break the intervention? 
Step 6 Sort post-its into meaningful categories 


(groups, in silence) and name them (groups) 


(How can the outcomes and activities be ordered in a logic model? 


Step 7 Work brainstorm opportunities and challenges that 
will make or break the intervention in this specific context 


g | 


= 


FIGURE 2.5 Revised version of the COP process 


Post-it notes (from step 5) into meaningful categories and providing them with a 
heading. We believe this addition may help the group also develop a shared under- 
standing about the context, further increasing the sense of being on the same page. 
The second revision is simply to explicate that the process also involves summariz- 
ing the findings in a program logic. 

In addition, we recommend that the following is also considered: 


e The time allowed for the process. In both cases, more than 2 hours would 
have been needed to allow more steps to be covered, and we would recom- 
mend others to allow at least 3 hours instead. Nevertheless, given the wealth 
of information earned on the current scale, the process does seem to be flex- 
ible enough to be valuable when time is more limited. 

e Care is needed in the formulation of the questions. We have also used this 
process in the context of stress management and in that case, the question 
needed to focus less on skills, attitudes, and knowledge and more on an imag- 
ined positive end state (imaging that one wakes up tomorrow and one’s work 
situation is in total balance: what would that look like?). 
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e In these cases, the researchers summarized the results from the COP process 
in a program logic, but this could also be done in collaboration with the 
organization. Yet, our experience is that people often perceive program logic 
to be a complex matter, and for the sake of using time and skills efficiently, 
it may be sufficient to have the interventionist summarize the information 
from the COP process in a logic model and then sense-checking it with the 
stakeholders. 
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